This invention relates generally to artificial neurons, which simulate the action of biological neurons, and more particularly, to neurons which operate by using a form of delta-sigma modulation to encode the neural states.
Generally, as shown in FIG. 1, an artificial neuron comprises a number of input signal paths, x.sub.0, x.sub.1, . . . x.sub.n, a signal weighting element, w.sub.0, w.sub.1, . . . w.sub.n, with a corresponding numerical weight value associated with each input signal; a plurality of multipliers 101 for combining the input signal x with the weighing element w to provide a weighted input signal xw; an adder 103, which combines the weighted input signals xw to produce an intermediate signal u; and an element which transforms the intermediate signal u (usually by a nonlinear function F) to produce the output signal y of the neuron. A general discussion on neural networks and neurons generally is found in Widrow, et. al., "30 Years of Adaptive Neural Networks: Perceptron, Madaline, and Backpropagation," Proceedings of the IEEE, Vol 78, No. 9, September 1990.
Artificial neurons may take many forms. Early studies were focused on neurons that operated entirely with continuous-time signals in the analog domain. More recently, neurons have been developed which use digital processing. In these digital neurons, the input and output signals typically are discrete-time, multibit digital values with the weights and computations stored digitally. Regardless of whether the neuron was analog or digital, the computation, including multiplication of the weights, addition of the weighted signals, and transformation by the nonlinear function, required a relatively complex system.
There have been several attempts to develop simpler data transfer technologies for use with neurons. In U.S. Pat. No. 5,170,071 to Shreve, the proposed artificial neuron is based on representing both the input signals and the weights as pseudorandom sequences, with a digital logic structure performing the computations. Counters within each of the variable weighting elements, which perform the weight multiplications, convert a common pseudorandom (noise) sequence into pseudorandom sequences with appropriate duty cycles. The system includes hardware for the backpropagation algorithm for training (i.e., determine the weights). This method does not use multiplexers or memories.
In U.S. Pat. No. 5,175,798 to Taylor, et al., input two-state signals are used that collectively form the address for identifying in a random access memory a numerical value, that then is used to convert a noise signal into a two-state signal with appropriate duty cycle.
The parameters of a neural processing network and of the interconnection pattern of the network and the weights of the neurons must be chosen for each particular application. The term "training" often is used to describe the selection of the parameters. If a network interconnection pattern is fixed, then training is performed by determining the appropriate weight values. Numerous methods for determining the appropriate weight values have been developed. The mathematical principles behind some of them depend on the smoothness of the transforming function F. A particular example of this is the backpropagation method, as generally described in Widrow, et. al, "30 Years of Adaptive Neural Networks: Perceptron, Madaline, and Backpropagation."
In order to simplify the system, the signal paths to and from the neuron could be configured as two-state channels, with the signal values encoded in some manner. One possible method is to encode the signal values to use each channel to pass multibit digital values in a serial manner. Another possible method is to use the signal pattern to encode the value by using the duty cycle of the signal for the value. This type of data transmission is commonly done in communications systems using delta modulation.
The neuron constructed in accordance with the preferred embodiment can be implemented, with relatively minor modifications, for use in either single bit or multibit networks. Either analog or digital circuitry can be used for the computations performed within the neuron. A central feature of the operation is conversion of data from the representational form used within the neuron to the form used outside the neuron (i.e, in the neural network). This conversion is achieved by using a delta-sigma modulator, or equivalent device, in the neuron itself to produce an output signal which has a certain psuedorandom character. Noise introduced by this process is pushed to high frequencies, so it is separated from the low frequency information-carrying signal. This separation of noise permits higher resolution of the information-carrying signal.